Relative Entropy Policy Search

This technical report describes a cute idea of how to create new policy search approaches. It directly relates to the Natural Actor-Critic methods but allows the derivation of one shot solutions. Future work may include the application to interesting problems. 1 Problem Statement In reinforcement learning, we have an agent which is in a state s and draws actions a from a policy π. Upon an action, it received a reward r (s, a) = Rsa and transfers to a next state s′ where it will do a next action a′. In most cases, we have Markovian environments and policies, where s′ ∼ p(s′|s, a) = Ps sa and a ∼ π(a|s). The goal of all reinforcement learning methods is the maximization of the expected return J̄(π) = E {∑T t=0 r(st, at) } . (1) We are generally interested in two cases, i.e., (i) the episodic open loop case where the system is always restarted from initial state distribution p(s0), and (ii) the stationary infinite horizon case where T → ∞. Both have substantial differences in their mathematical treatment as well as their optimal solution. 1.1 Episodic Open-Loop Case In the episodic open-loop case, a distribution p(τ) over trajectories τ is assumed and a return R(τ) of a trajectory τ , both are given by p(τ) = p(s0) ∏T t=1 p(st+1|st, at)π(at|t), (2) R(τ) = ∑T t=0 r(st, at). (3) The expected return can now be given as J̄(π) = ∑ τ p(τ)R(τ). Note, that all approximations to the optimal policy depend on the initial state distribution p(s0). This case has been predominant in our previous work.